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Abstract. The present study aims to reveal English as a Foreign Language (EFL) 
learners’ online learning behaviour in a cross-platform web application for EFL 
vocabulary learning courses at a national university in Japan. Students were 
divided into two groups: those who were provided with the new, cross-platform 
application for vocabulary learning, and those with the older, PC-based Web-Based 
Training (WBT) system. The learning contents, learning processes, and the course 
evaluation systems were almost identical between these two groups. The log data 
of approximately 850 students over a year was extracted, and the online learning 
behaviour was statistically compared between the two groups. The comparison 
results can be summarised as (1) the total learning duration, the outcome, and 
learning efficiency are almost equivalent across the groups, and (2) the students with 
the new application exhibited a relatively significant tendency of frequent, steady, 
and periodical logins than those with the old one. The analyses suggested that the 
cross-platform, mobile-optimised web application elicited the students’ ability to 
regulate their everyday self-accessed online learning. 
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1. Introduction 


Enokida et al. (2017) reported on the development of the web application to be 
utilised for EFL vocabulary learning courses at a national university in Japan. The 
older version of the WBT system, based on Adobe Flash, has long been used since 
these courses were first implemented in 2011, targeting 1,000 first-year students 
at the university. In the academic year 2017, the Flash-based system was partially 
replaced by a new, HTML5-based one: a group of students were provided with the 
new system, while the rest of them did their coursework on the old one. 


This division has provided two data sets with which the online learning behaviour 
of the two groups can be compared. Although there has been a steady stream of 
studies that have reported the impacts of two different CALL materials on students’ 
usability feedback (e.g. a questionnaire survey) and learning outcomes (e.g. pre- 
and post-test scores), relatively few studies have examined students’ online learning 
behaviour itself. Considering the positive effects of mobile English vocabulary 
learning on the learning performances and interests of learners (Chen & Chung, 
2008), it is expected that providing a cross-platform application will positively 
affect their online learning behaviour as well. The present study extracted online 
learning log data of approximately 850 students over one year, and statistically 
compared the students’ online learning behaviour between the two groups. The 
purpose of the present study is to answer the following research question: 


¢ How can users of the new, cross-platform application be compared with 
those of the older, PC-based WBT system in terms of their learning 
behaviour, as measured by (1) the frequency (number) of logins, (2) 
the periodicity (cycle) of logins, (3) the outcome, and (4) the learning 
efficiency? 


2. Method 


2.1. Participants and instruments 


There were 841 participants in this study who took the semester-based online 
courses, ‘Communication Basic I/II’ in academic year 2017, which are compulsory 
courses for first-year students at Hiroshima University. They were required to learn 
3,000 words per semester (15 weeks) and 6,000 per year on a self-study basis 
through the WBT system, the HiroTan App. The materials are based on an essential 


48 


Tracking online learning behaviour in a cross-platform web application... 


vocabulary list called The Hirodai Standard 6000 Vocabulary List, which was 
specially developed for the courses. 


2.1.1. The Hirodai Standard 6000 Vocabulary List 


The Hirodai Standard 6000 Vocabulary List consists of two levels (basic and 
advanced) of words that are used in daily communication as well as in academic 
and business contexts. The basic list consists of 2,000 words and the advanced 
1,000 words — all to be learned within one university semester. Each word comes 
with its Japanese translation and a sample sentence with its Japanese translation. 
The students were provided with a printed textbook and downloadable audio 
podcasts with which they could listen to the pronunciation and sample sentence of 
each word, as well as the WBT materials. 


2.1.2. The HiroTan App (WBT system) 


An original WBT system is used for these courses to enable the students to learn the 
forms, meanings, and pronunciations of the large number of words in small steps 
(10 words per unit) and review them repeatedly (Enokida et al., 2017). Detailed 
log data is stored in the server, so that it can be analysed to assess the students’ 
online performance. The latest cross-platform version of the WBT system is called 
HiroTan (see Figure 1). 


Figure 1. The HiroTan App: learning ten words in a unit 
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2.2. Data collection 


The participants engaged in a self-access vocabulary list learning for one academic 
semester, but reflecting the above-mentioned transition from the old to the new 
system, they were divided into the two groups based on the accessibility conditions: 
(1) a Flash-based application via PC and PC web browsers, and (2) a newly 
developed HTMLS-based application that is optimised to mobile devices such as 
smartphones and tablets. The learning contents, learning processes, and the course 
evaluation systems were almost identical between these two groups. 


2.3. Data analysis 


The log data of the students’ 120-day-long learning behaviour that was automatically 
recorded by the implemented function in the two accessibility conditions were 
then extracted, in order to compare (1) the frequency of logins in a day, (2) the 
periodicity of logins, and (3) the number of the contents completed during the 
semester. Since the size of the log data is too big to perform statistical testing, the 
present study visualised the data and fitted some statistical distributions using the 
maximum likelihood estimation as appropriate. 


3. Results and discussion 


The present study at first compared the frequencies of logins in a day, which are 
visualised in Figure 2. The number of students in each data set was standardised as 
density. The figure clearly shows that the new version (left) promoted more frequent 
logins than the older (right), meeting our expectations. To support this statistically, 
the present study fitted the exponential distribution using the maximum likelihood 
estimation, and compared the parameter 1 for each group, since the data are, at a 
glance, extremely skewed. The estimated parameter values are A=0.039 for the 
new HTML5S-based application, and A=0.059 for the old Flash-based application, 
and the expected values were calculated by the parameters. The values were 25.64 
logins in a day for the new version, and 16.90 for the old version. 


Secondly, the present study inspected the periodicity of logins. Figure 3 represents 
the time series of the cumulative total number of logins in a day for both the 
conditions. There are higher numbers of daily logins to the new version throughout 
the 120 days of the online course, which means that the new version has a strong 
tendency of shorter cycles of logins than the older. This tendency becomes clearer 
when we break down the data into days of week. As Figure 4 shows, the new 
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version has a statistically steady half-week cycle of logins (Day 1/7 and Day 4) 
while the older does not exhibit the tendency. 


The total numbers of the contents completed during the semester were almost 
equivalent between the two groups. That makes sense considering that the 
minimum requirements of the materials to be learned to pass the online course 
were the same in both groups. Also, against the two previous findings that suggest 
the new versions’ advantages, no striking differences were found in the outcome 
of the courses, which were measured by the vocabulary tests, and the total learning 
duration, which can be operationalised by the logout — login time difference. That 
might imply that the outcome and the learning efficiency themselves were not 
facilitated by the replacement of the systems. 


Figure 2. Histograms representing the cumulative total number of students’ login 
to the web apps in a day 
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Figure 3. Time series plots representing the total number of logins in a day 
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Figure 4. Plots representing the total numbers of logins in a day, showing the 
differences of logins by the day of week; Day | represents Sunday and 
Day 7 is Saturday 
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4. Conclusions 


The analyses of the online learning behaviour can be summarised as (1) the total 
learning durations, the outcome, and learning efficiency are almost equivalent 
across the groups, and (2) the students with the new, cross-platform application 
exhibited a relatively significant tendency of frequent, steady and periodical 
logins than those with the old, PC-only one. The analyses suggested that the cross- 
platform, mobile-optimised web application elicited the students’ ability to regulate 
their everyday self-accessed online learning. 


Since only the new application is currently used in the Communication Basic 
courses, our future directions will include: (1) comparing the online learning 
behaviour of successful vocabulary learners with that of unsuccessful learners, 
and (2) addressing the questions regarding the effectiveness of WBT-based list 
learning, as given in Enokida et al. (2017). 
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